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ABSTRACT 

Recommender Systems (RS) have been applied in the Technology enhanced Learning (TeL) field for facilitating, among 
others, Learning Object (LO) selection and retrieval. Most of the existing approaches, however, aim at accommodating 
the needs of learners and teacher-oriented RS are still an under-investigated field. Moreover, the systems that focus on 
teachers, do not explicitly exploit their ICT competence profiles when providing LO recommendations. This is a 
significant drawback, since it can result in LO recommendations that are beyond the teachers' competence to use. 
Towards tackling this issue, this paper extends previous work and proposes a teacher ICT Competence -based RS that 
considers teachers' ICT Competence Profiles when recommending Learning Objects. Moreover, the results of its 
accuracy evaluation are presented. The results indicate that the proposed approach provides high predictive accuracy and 
outperforms commonly used, existing RS approaches. 
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1. INTRODUCTION 

Recommender Systems (RS) are adaptive software applications aimed at providing suggestions for 
potentially useful items to the users (Bobadilla et al., 2013). They are widely used in many contexts, such as 
Movie Industry (Bobadilla et al. 2011) and Technology Enhanced Learning (TeL) (Manouselis et al. 2013a). 
RS are mainly divided in three categories in terms of the method applied for providing these 
recommendations (Ricci et al. 2011). These are (a) content-based recommenders, which build user models 
based on the items that users have already interacted with, (b) collaborative filtering recommenders, which 
produce recommendations for users based on the rating patterns of the users' neighbors (like-minded users), 
and (c) hybrid recommenders, which combine techniques from both previous approaches in order to reap the 
benefits of both, while tackling their individual drawbacks. 

In the context of TeL, one of the key applications of RS is the location and selection of Learning Objects 
(LO) (Manouselis et al. 2013b). The majority of existing TeL RS, however, is focused on the target group of 
learners and aims to accommodate their specific characteristics in the recommendation process, such as their 
learning styles or their learning preferences (Ferreira-Satler et al. 2012; Chrysafiadi & Virvou 2013). From 
the standpoint of teachers, however, less research has been performed and few approaches actually consider 
teacher characteristics towards providing more personalized recommendations for their course design and 
delivery (Sergis et al., 2014). Due to the diversity in these characteristics, the "appropriateness" and 
usefulness of each LO to-be-recommended is different for each teacher based on her unique profile and 
needs. A significant type of such characteristics are the teachers' ICT Competences, since the latter are an 
important factor affecting their everyday practice (Sang et al. 2010; Goktas et al. 2013). Therefore, 
accommodating for these teacher attributes has the potential to provide significant added value in the teacher- 
oriented RS. 

Under this light, previous work was performed towards tackling the identified research problem of 
facilitating LO recommendation for teachers based on their ICT Competence profiles (ICT-CP). This work 
included the design and evaluation of a set of mapping rules linking teachers' ICT Competences to LO 
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metadata elements for identifying appropriate LOs for each teacher, competence-wise (Sergis et al., 2014). 
The contribution of the present paper is to extend this work by designing and developing an ICT 
Competence-based RS, which utilizes the proposed mapping rules and delivers informed LO 
recommendations to teachers based on their ICT-CP. Moreover, the predictive accuracy of the proposed 
approach is benchmarked against commonly used user- and item-based collaborative filtering approaches. 

The remainder of this paper is structured as follows. In Section 2 the current state of the TeL teacher- 
oriented RS landscape is presented. Moreover, the research problem and the contribution of the present paper 
is clearly identified. Section 3 presents the proposed approach for providing ICT Competence-based Learning 
Object recommendations to teachers. Section 4 comprises the evaluation method and results for validating 
the proposed approach's predictive accuracy. Finally, Section 5 contains the conclusions drawn and the future 
work in this research agenda. 


2. PREVIOUS WORK: TEACHER-ORIENTED TEL RECOMMENDER 
SYSTEMS 

In previous work (Sergis et al., 2014), a literature review was performed in the landscape of TeL 
Recommender Systems in order to highlight approaches that were addressed to teachers. The results of this 
process identified a set of 22 teacher-oriented RS. Moreover, an additional refinement of this list highlighted 
those approaches that created profiles with personal characteristics of the teachers (in addition to simple 
social data profiles) and utilized these data in order to drive more informed recommendations. This list 
comprises 7 RS and is presented in Table 1. Additional information depicted in Table 1 include the type of 
profile data that these RS capture. 

Table 1. Existing teacher-oriented TeL Recommender Systems 

# Paper Profile 

data 

Demographic data 

User work-related knowledge levels - not restricted 
to teachers 
Not clearly stated 

User interest on items based on social data and item 
descriptions 

Users experiences of resource use - not restricted to 
teachers 

Not clearly stated 

Demographic data, Information Technology 
experience - Low metric granularity 
As Table 1 depicts, each of the seven identified RS utilized a different type of user data in order to create 
their profile, while some did not include explicit information on the manner in which the reported profiles 
were created. However, a commonly identified issue in all approaches was the universal lack of consideration 
for exploiting the teachers' ICT competences. Bearing in mind the significance of these characteristics to the 
level of the level of ICT uptake of teachers (Vanderlinde et al. 2014), their integration in the LO 
recommendation process was proposed as a significant research problem. 

More specifically, a research problem was formulated, namely how can the process of LO 
recommendations to teachers based on their ICT-CP be facilitated. This is reinforced by the reported need for 
more informed user modeling processes towards better recommendations (Adomavicius & Tuzhilin 2005). A 
first step towards tackling this issue was performed by constructing and evaluating a set of mapping rules 
linking (a) specific ICT Competences as they are depicted in the UNESCO ICT Competency Framework for 
Teachers (http://tinyurl.com/ocaxgqk) (ICT-CFT) to (b) specific Learning Object Metadata attributes, as they 
were depicted in the OpenDiscoverySpace (ODS) Project (http://www.opendiscoveryspace.eu) LOM 
Application Profile (Sergis et al., 2014). This work showed that the proposed mapping rules provided high 
accuracy and could, therefore, effectively classify LOs as appropriate in terms of each teacher's ICT-CP. 


1 Bozo et al. (2010) 

2 Schoefegger et al. (2010) 

3 Sielis et al. (2012) 

4 Schirru et al. (2010) 

5 Aehnelt et al. (2008) 

6 Cobosetal. (2013) 

7 Zapata et al. (2013) 
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The next step in this research agenda is the investigation of the potential added value that this approach 
could deliver in the teacher-oriented RS. Therefore, the aim and contribution of the present paper is to 
progress this research agenda by reporting on the process of designing, implementing and evaluating a RS 
that utilizes these mapping rules in order to deliver ICT Competence-based LO recommendations to teachers. 
More specifically, the RS overview is presented and its recommendation methodology is described in detail. 
Finally, the predictive accuracy evaluation of the system is presented and the added value of this approach is 
discussed. 


3. ICT COMPETENCE-BASED LEARNING OBJECT RECOMMENDER 
SYSTEM FOR TEACHERS 


The overview of the proposed ICT Competence-based approach to recommending LOs to teachers is 
presented in Figure 1. 
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Figure 1. Overview of proposed teacher ICT Competence-based Learning Object Recommendation System 

The methodology for creating LO recommendations includes an initial pre-filtering of the full set of LOs, 
in order to keep only those that correspond to the teacher's ICT-CP. This step is realized by utilizing the 
mapping rules designed and evaluated in previous work (Sergis et al., 2014). At the end of this process 
(which is performed offline), a refined pool of LOs has been created that contains the "appropriate" LOs for 
the active teacher. Following this step, this refined pool of LOs is used for delivering the recommendations 
based on the ICT Competence-based neighbors of the active user. Therefore, the proposed approach utilizes 
the teachers' ICT-CP in a twofold manner, namely (a) as a pre-filtering mechanism for identifying LOs that 
are beyond the active teachers' competence to use, and (b) as an alternative neighborhood creation method, 
focusing on ICT competence profile similarities instead of common rating patterns. 
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As the Figure 1 describes, the proposed approach comprises a set of distinct steps, each of which are 
presented in the following sections. 


3.1 Teacher ICT Competence Profile Depiction 

As aforementioned, the teacher ICT-CP was depicted using the UNESCO ICT Competency Framework for 
Teachers (CFT). The reasons for selecting the CFT over alternative frameworks have been discussed in 
Zervas et al. (2014).This framework addresses all aspects of a teacher’s practice within their work context 
and aims to provide comprehensive teacher ICT competency standards for assisting them in successfully 
addressing all relevant dimensions. It comprises 6 categories of ICT Competences spanning 3 proficiency 
levels. Each competence category is further divided in sub-competences that are differentiated according to 
the proficiency levels. The total number of sub-competences described in the CFT is 61. Therefore, a teacher 
ICT Competence Profile submitted to the system would include each of these sub-competences, evaluated in 
terms of whether the active teacher possesses it or not. 

3.2 ICT Competence-based Neighborhood Creation 

The neighborhood creation process in the proposed approach is performed in a novel manner. More 
specifically, the method employed in the proposed approach utilizes the teachers' ICT-CP in order to create 
these neighborhoods. This alternative has the potential to provide more informed recommendations, since 
users (in our case, teachers) would receive LO recommendations based on the opinions of colleagues that 
have the same competence background with them. 

In the context of this paper, the ICT Competence-based neighborhoods were constructed using the 
Jaccard co-efficient (JC) (Verbert et al. 2011). This coefficient measures the similarity between two sets (in 
our case ICT Competences) by dividing the number of common items with the number of different items 
from both sets of rated items. It is calculated using the formula: 

U n n / F | 

!C ~ I /„l+ I/ V .| - l/tf n ly I 

where and I F are the Competences of users U and V respectively. 

Within the context of the proposed RS, the output of this formula served as an indicator of the "closeness" 
between two users, i.e., how similar are their ICT Competence profiles. This indicator was utilized in the 
recommendation process of the proposed approach (section 3.4) as a weighting factor for predicting the 
rating of the active user based on the ratings of her neighbors. 

3.3 Pre-filtering Learning Objects 

The second step in the proposed ICT Competence-based recommendation system relates to the pre-filtering 
of the available LOs in order to consider only those that are "appropriate" for the active teacher, in terms of 
her ICT-CP. This process was performed by utilizing the set of mapping rules linking specific ICT 
Competences described in the CFT and specific LO metadata elements. Therefore, for each LO the 
corresponding mapping rule was fired and the system would flag the LO appropriately in terms of the active 
teacher's ICT-CP. At the end of this process (which was performed offline to reduce the systems' response 
time), each teacher would have their own unique set of "appropriate" LOs, based on their ICT-CP. Finally, 
utilizing this refined pool, the system would deliver the LO recommendations. 

3.4 ICT Competence-based Learning Object Recommendations 

The process of delivering the LO recommendations is similar to the existing collaborative filtering 
approaches. More specifically, a weighted average of the neighbors' ratings was utilized in order to predict 
the rating that each user would assign to a LO that was to-be-recommended. The difference with existing 
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collaborative filtering approaches, apart from the fact that the candidate LOs would come from the refined 
pool of "appropriate" LOs instead of the full pool of all available LOs of the repository, is the nature of the 
weights assigned to the weighted average metric. More specifically, as aforementioned in a previous section, 
these weights were the output values of the Jaccard coefficient used for creating the competence-based 
teacher neighborhoods. Therefore, "closer" neighbors, i.e., neighbors with more similar ICT-CPs to the active 
user and higher Jaccard coefficient, had a greater impact in the recommendation process. 

The predicted rating for each LO i by the active user u was calculated based on the formula: 

Ei w j * »J: 
rui_ S>j- 

where ?j, is the rating provided by neighbor j on the LO i, j is the j th neighbor of the active user and N is 
the neighborhood size utilized. Based on the generated predicted rating, the system could infer not only if 
each LO would be appropriate for the active user, but, moreover, whether it would be of interest to her. 

The proposed system was evaluated in terms of its predictive accuracy. The evaluation methodology 
followed is described in the following section. 


4. EVALUATION 


4.1 Dataset 

An experiment was performed towards the evaluation of the proposed recommendation method based on 
teachers' ICT Competence profiles. The dataset which was utilized came from the OpenDiscoverySpace 
(ODS) Project (http://www.opendiscoveryspace.eu). The reason for selecting this dataset was that it 
contained existing ICT Competence profiles provided by 115 real teachers. The overview of the retrieved 
dataset is presented in Table 2. Aggregation Level 1 LOs refer to standalone Educational Resources (e.g., 
flash simulations), while Aggregation Level 2 LOs refer to Lesson Plans. 


Table 2. ODS Dataset Overview 


Category 

Sample Size (N) 

Rating Data 

User ICT-CP 

115 

- 

AL 1 Learning Objects 

523 

1375 

AL 2 Learning Objects 

475 

986 


4.2 Evaluation Method 


The evaluation was based on the predictive accuracy of the proposed approach. It involved the standard 
"Leave-N-out" technique, where 70% of the data are used as a "training" set for the recommender system and 
the remaining 30% of the data are used as a "test" set for evaluating the performance of the system; in our 
case, its predictive accuracy. The Root Mean Squared Error (RMSE) metric was selected for measuring the 
predictive accuracy of the system (Shani & Gunawardana 2011). This metric is calculated based on the 
formula: 


RMSE = 



Oui - fui) 3 

11, LET 


where r ui is the set of generated predicted ratings, r ui is the set of known ratings and T is the set of users 
and items for which the ratings are known. 

Moreover, the proposed approach was benchmarked against two "control" recommendation methods, 
namely a user-based collaborative filtering approach and an item-based collaborative filtering approach. Both 
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these approaches utilized the Pearson correlation coefficient (PCC) for creating the user/item neighborhoods 
(Verbert et al. 201 1). The Pearson correlation coefficient is calculated based on the formula: 

_ ^LEJ^LLL ~ 

- r lt ¥ S ieJ (r wi - r K ¥ 

where / is the set of items that both users u and vr have rated, r^- and denote the ratings of the users u 
and vv on the item i A while ^ and ^ denote the average ratings of the two users respectively. Following this 
step, both "control" approaches utilized the PCC as a weighting factor for creating the predictions of the 
active user for the LOs that they had not yet rated. 

Finally, the evaluation experiment was run for an increasing number of neighborhood size in order to 
monitor the behavior of each approach in each occasion. The results of the evaluation experiment are 
presented in the following section. 

4.3 Evaluation Results 


The results of the evaluation experiment are presented in Figure 2 and Figure 3. The former presents the 
results concerning Aggregation Level 1 LOs, while the latter presents the results for Aggregation Level 2 
LOs. The reason for examining both these LO Levels is that in the "Pre-filtering" step of the proposed 
approach, different mapping rules are utilized for these LO Levels, and therefore, both had to be evaluated 
compared to the existing approaches. 



-User-based PCC 
Item-based PCC 
-Competence -based JC 


> T eig lib oi hood Size 


Figure 2. RMSE results for Aggregation Level 1 Learning Objects 

As the Figures 2 and 3 depict, the results of the evaluation experiment are very promising. More 
specifically, the proposed approach achieved high levels of predictive accuracy (in the form of low RMSE 
value) for both Aggregation Level LOs. This means that it can effectively identify LOs as appropriate or not 
in terms of the teachers' ICT Competences, as well as to select those that are interesting to her. 

Moreover, compared to the user- and item-based approaches, a significant improvement was detected. 
This means that incorporating the teachers' ICT-CP in the LO recommendation process (and the manner in 
which this was performed) can provide added value in existing RS systems and offer teachers with more 
useful services. 

Another important finding is that ICT Competence-based recommendations out-perform existing 
commonly used collaborative filtering approaches for all neighborhood sizes, signifying that they could 
prove beneficial in tackling some common collaborative filtering problems, such as the new user problem 
(Bobadilla et al. 2012). 
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Figure 3. RMSE Results for Aggregation Level 2 Learning Objects 

The new user problem occurs within RS when it is not possible to make reliable personalized 
recommendations due to an initial lack of ratings by the active user. This is more evident in educational 
datasets, which are usually very sparse. The results of this paper show that the inclusion of the teacher profile 
information as a means for creating neighborhoods could assist in remedying for this issue in a significant 
degree, since the active user could be linked to a set of appropriate neighbors without the need for their own 
ratings. 

The above overall findings provide with strong evidence that the inclusion of teachers' ICT-CP in the 
learning object recommendation process within Learning Object Repositories (LORs) can provide enhanced 
personalized services in a number of ways and can accommodate each teachers' individual profile and 
competence-related needs. 


5. CONCLUSIONS AND FUTURE WORK 

This paper, building on previous work, proposed a system that integrated the teachers' ICT-CP in the LO 
recommendation process within LORs. This work was focused at tackling a research gap in the RS 
landscape, related to the lack of consideration of the teachers' ICT-CP when recommending LOs. This was 
considered as a significant drawback, bearing in mind the apparent importance of these personal 
characteristics in the everyday level of ICT uptake of teachers. 

Towards measuring the level of the added value of the proposed approach, an evaluation experiment was 
performed, which focused on benchmarking the predictive accuracy of the approach against two commonly 
used collaborative filtering approaches. The results provided with evidence that the proposed approach can 
deliver better recommendations (in terms of low system RMSE) for all LO Aggregation Levels and can also 
perform adequately in small neighborhood sizes. 

Bearing in mind the added value that the proposed approach showed to deliver in the LO recommendation 
process for teachers, future work should be focused on identifying a robust framework for eliciting these 
teacher characteristic from their usage patterns within LORs. The need for this springs from the fact that 
teachers (and users in general) are usually either unwilling to provide personal information or when they do, 
the validity of the provided data cannot be ensured (Belk et al. 2013). Therefore, in order to overcome the 
above issue, and reap the reported benefits of ICT Competence-based LO recommendations, mechanisms 
should exist for inferring the teachers' profile from their usage patterns (Marin et al. 2014). 
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